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Test Reviewing: Problems and Prospects 

David J. Weiss 
University of Minnesota 
Introduction 



A comprehensive test reviewing process is desirable for at least three 
reasons. First, test reviews provide valuable Information to test users in 
terms of selection, use, and interpretation of data from psychological in- 
struments. Secondly, test reviews are a means of providing information to 
test authors and test publishers about the desirable and undesirable charac- 
teristics of the instruments they publish. As a result of this kind of in- 
formation, test publlslinrs and authors may improve Instruments with subse- 
quent benefit to the users of psychometric Instruments, A third Important 
reason for the existence of test reviewing programs, is to provide a vehicle 
for the exercise of professional Influence over the kinds of Instruments 
published for use in the provision of psychological services, and in psycholo- 
gical research. While many teat reviews are not directly aimed at meeting 
the ethical responsibilities of the profession, they may have an indirect 
effect by providing relevant information to both test users and test publi- 
shers. 

By far the vast majority of teat reviews have been published by Oscar 
Buros in the Mental Measurements Yearbooks (MMX) series. In the thirty years 
during which Professor Buros has published six tWo (a seventh is now in 
preparation) he has critically reviewed virtually every published psycholo- 
gical test, Inventory, and questionnaire. A small minority of test reviews 
appears in various psychological Journals, with the number of reviews be- 
ginning to show an Increase in the last few yesrs. Hcwever, because of 
pressures on journals for publishing other types of materials, test reviews 
have not been given a prominent position in the press for space in current 
Journals. 

In 1965, an ad hoc committee of the Division of Evaluation and Measure- 
ment (Division 5) of the American Psychological Association was convened to 
determine whether current test reviewing procedures were meeting the needs in 
the field. After two years of discussion of the problem with leading indivi- 
duals in measurement, education, and related fields, the Division 5 committee 
felt that tlte formation of a more permanent vehicle to coordlnete test review- 
ing activities would be appropriate. The committee then contacted all rele- 
vant organisations and held a series of meetings with Interested organisations 
to discuss the possible direction of a coordinated effort in the field of 
test reviewing. These discussions lead to formation of the Inter-Assocletlon 
Council on Test Reviewing (IACTR), which was formally established in 1967* 

The purpose of IACTR is "to promote and facilitate the dissemination of 
information about and critiques of published psychological and educational 
tests, including devices and materials for evaluation and prediction," 

IACTR Is composed of representatives of a number of organisations with 
Interests in the use of psychological and educational assessment devices. 
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Since Its founding, ZACTR has been concerned with studying the problem In 
more detail, to arrive at a permanent solution to what appeared to be the 
problems in the area of test reviewing. At its first annual meeting in 
1968, IACTR formulated plans for a survey of test users and Journal editors 
to further clarify the problems In test reviewing and to obtain the reactions 
of test users to some of the prospective solutions to these problems. This 
report is based on the findings of these surveys. 

The Surveya 



Test Users Survey 

A "test user" was defined as a member of an organisation which was con* 
conerned with use of psychological teste and assessment devices In Indivi- 
dual assessment or research. Bleven organisations wore selected using this 
criterion. The names of the organisations and the abbreviations used in 
this report are shown In Table 1. All questionnaires were mailed In the 
spring of 1969. 



The most recent membership lists were obtained for each of the 11 
organisations. Within each membership list, 216 names were selected by 
random procedures. Bach person so selected was sent a questionnaire and 
a cover letter deacriblng the purpose of the survey. The questionnaire 
asked a number of questions concerned with specific problems and prospects 
in the field of test reviewing. It also Included a demographic data page 
designed to obtain Information to describe the samples from which the data 
were obtained. Descriptive characteristics of the respondents are shown 
In appendix Tables A-l through A-4 for the total group and separately for 
each or gen lest Iona 1 subgroup. Appendix Tables A-J and A-6 give Information 
on the kinds of assessment instruments used by respondents. 

Insert Table 1 about here 



Table 1 shows the percentage of returns by group, end the percent of 
total group represented by each of the 11 organisations. As Table 1 indi- 
cates, the lowest percentage of returns (39.4%) was from the Division of 
Clinical Psychologists (Division 12) of the American Psychological Associa- 
tion (APA). Divisions 15 and 16 of AFA also had relatively low returns. The 
two organisations with the highest rate of returns were the Association for 
Measurement and Evaluation in Guidance (AMEG) with 61.1% responding, and the 
national Vocational Guidance Association (KVGA) with 60.2% responding. 
Accordingly, these latter two groups represented over 10% of the total 
analysis group of 1,147. respondents. 

Two groups whose members were elso likely to have an Interest in psycho- 
logical testing were not specifically sampled in this survey. These were 
the National Council on Measurement in Education (NCMR) and Division S of 
the American Psychological Association. However, a cross tabulation of 
organics tlonal membership for those organisations sampled with all organi- 
sations ssmpled with the addition of NCHB and Division 5 of APA, showed that 
a portion of the membership of each of those organisations was Included in 
the survey. This cross- tabulation is shown in Appendix Table A-7. 
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Table 1 



Percentage distribution of respondents by organization sampled, 
end percent responding in each group 



Organization 


Abbreviation 


N 


Percent 
of Total 
Group 


PSrceotL' 

Respond* 

ing 


American Educational Research 
Association, Division D 


ABRA, Div. D 


112 


9.8 


54.4 


American Rehabilitation 

Counseling Association 


ARCA 


103 


9.0 


47.7 


American School Counselors 
Association 


ASCA 


103 


9.0 


47.7 


Association for Measurement and 
Evaluation in Guidance 


AMEG 


132 


11.5 


61.1 


National Vocational Guidance 
Association 


NVGA 


130 


11.3 


60.2 


Society for Projective Techniques and 
Personality Assessment 


SPTPA 


107 


9.3 


49.5 


American psychological Association 
Division 12, Clinical Psychology 


APA, Div. 12 


85 


7.4 


39.4 


Division 14, Industrial Psychology 


APA, Div. 14 


98 


8.5 


45.4 


Division IS, Educational Psychology 


APA, Div. 15 


90 


7.8 


41.7 


Division 16, School Psychology 


APA, Div. 16 


87 


7.6 


40.3 


Division 17, Counseling Psychology 


APA, Div. 17 


100 


8.7 


46.3 


Total Group 




1,147 


«e« 


43.3 
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As Table A- 7 shows, 11.9% of the total group were menders of NOME, and 8.8% 
of the total group were members of Division 5 of APA. These figures agree 
reasonably well with the percentage composition of the total group by groups 
actually sampled (see Table 1). Thus, these groups were well represented 
in the total study group. 

However, Table A* 7 also shows that other groups which were evenly sam- 
pled in the research design were dlsproportlonally represented in the total 
group as a result of overlapping group memberships. Specifically, the 
National Vocational Guidance Association (NVGA) was represented by 33.8% of the 
total group of 1,147 respondents. This compares to only 9.9% members of Division 
14 of APA members (Industrial) since very few Division 14 members held 
duplicate membership in other organisations studied. The results for the total 
group should be Interpreted with a consideration of these disproportionate 
frequencies. 

Journal Editors Survey 

A separate questionnaire was also sent to Journal editors. Some of the 
questions in the Journal editor's survey were identical to those used in the 
test user survey. Additional questions were asked of Journal editors to 
obtain specific information concerning their views on certain problems not 
relevant to the test user audience* 

A total of 28 questionnaires were mailed to the editors of Journals 
identified as publishing test reviews, or publishing research using psycho- 
logical and educational assessment devices. Of the number mailed, returns 
were received from 21 Journals, representing a 75% return from the total 
group mailed. Appendix B lists those Journals whose editors cooperated in 
this survey. 

Analysis 



Analysis of the data was based on the computation of percentage of 
respondents choosing each of the multiple choice alternatives in the question* 
nalres. Some questions asked respondents to respond in a categorical "yes" 
or "no" response; for these questions the percentage "yes" and the total 
number responding is reported. In addition, for ths "yea-no" questions, 
several questions asked a relative question in terms of which of several 
activities was most important or preferred by the respondent. 

Because of the possibility of there being group differences in percen- 
tage responding, the analysis also Included the computation of '.hl-square 
tests for significance of the differences among the 11 groups. For those 
questions which were the same in both the Journal editor and the teat usar 
survey, the results of the Journal editor survey are presented in comparison 
with the results for the teat users. However, because of the small number 
of Journal editors replying to the survey, the results for the Journal 
editors were not Included in the computation of chi-square teats of group 
differences in response frequencies. 
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Analysis of data was facilitated by a grant of subsidized computer time 
from the University of Minnesota Computer Center* 

Problems in Test Reviewing 



Number of test reviews 



A question in both the journal editors and test user survey asked "How 
do you feel about the number of test reviews being published?" Table 2 
shows the percentage distributions of responses on the five-point rating 
ocale provided for this question. Of the total group of 1,057 individuals 
who responded to this question, only 18*5% indicated that there was “no. in- 
crease required" in the number of test reviews being published. Only 4.3% 
of the total group indicated that "less" test reviews were appropriate. 
Combining the three response alternatives indicating that an increase in the 
number of test reviews would be desirable, 77% of the total group of respon- 
dents indicated that more test reviews wero appropriate. 

Insert Table 2 about here* 

An analysis of the group differences in response to this question, 
among the 11 groups studied, yielded a chi-square value of 54.44, with an 
associated probability level (40 df) of between .10 and .05. This suggest- 
ed a tendency for the distributions of the responses of the groups to be 
different. While about 50% of each of the 11 groups indicated that "more" 
test reviews are needed, 7*6% of the members of the Society for Projective 
Techniques and Personality Assessment indicated that "less" test reviews 
are needed, compared to only 1*1% of the members of the American School 
Counselors Association. In terms of the response of "need many more" test 
reviews, responses varied from 7.7% for members of APA Division 16 to 19.8% 
of AERA Division D members* 

Among Journal editors, 65% of the 20 respondents indicated that "more" 
test reviews were needed, and an additional 15% responded in each of the 
categories "need many more", and "need a slight increase." In total, 95% 
of the Journal editors supported the need for some increase in the number of 
test reviews. None of the Journal editors indicated that less test reviews 
were needed* 

The consensus of both journal editors and test users, then, is that more 
test reviews are needed* Hence, while Buros^ Mental Measurements Yearbooks 
provide a large number of comprehensive test reviews, almost 8 out of 10 
potential and actual test users feel the need for an increase in the number 
of test reviews. 

Nature of test reviews 



Two questions in the test users survey were concerned with the nature of 
test reviews. Question 2 asked "How do you feel about the nature of ourrent test 
reviews?"; respondents were asked to reply on a four-point scale from "much too 
technical" to "not technical enough". A related question (question 4) asked 




Zable 2 

Percentage distribution* for Question 1: Bow do you feel about the number of test reviews being published? 
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"To what extent do current test reviews meet your needs?"; respondents were 
asked to characterize test reviews as "too technical"* "adequate", or "too 
practical". The second question was also asked of journal editors, in terms 
of how well test reviews meet the needs "of test users". 

For the first of the two questions (see Table 3), about half of the 
total group of respondents felt that the nature of current test reviews is 
"about right"; 51% of the total group of respondents chose this alternative. 
However, 31*5% of the total group indicated that test reviews were either 
"too technical" or "much too technical". Only 17*4% indicated that current 
test reviews are "not technical enough". 



Insert Table 3 about hero 

Analysis of the organizational sub-group differences in response to 
this question yielded a value of chi-square significant well beyond the 
.01 level of statistical significance. This indicated wide differences 
among the meobers of the 11 organisations in terms of their responses to 
this question. In general, tbo members of ABA, the Society for Projective 
Techniques and Personality Assessment, and ABBA Division D, tended to indi- 
cate more than the other groups, that test reviews were "not technical 
enough". Por these groups, the percentage choosing the latter response 
varied from 20.5% to 26.3%. Among the groups representing the American 
Personnel and Guidance Association (ARCA, ASCA, AMEG and NVCA) the 
percentage indicating that current test reviews were "not technical 
enough" varied from a low of 6.3% to 10.5%. 

A similar contrast is shown in the percentages of respondents indica- 
ting that test reviews were "too technical" or "much too technical". Only 
15.1% of the members of the Division of industrial Psychologists of APA 
(Division 14) indicated that test reviews were either "too technical" or 
"much too technical". This contrasts with a total of 50.5% of the ARCA 
members and 53. 7X of the ASCA members choosing these alternatives. The 
percentage of individuals indicating test reviews were "much too technical" 
varied from 0% for members of APA Division 14, go 12.1% of ARCA members. 

These results, indicating large and statistically significant differ- 
ences among the members of various organizations, suggest that test reviews 
need to bo written at various levels of technical sophistication. It 
would appear that members of the more applied groups such as ARCA, ASCA, 

AMB0 and NVGA require test reviews at a lower level of technical sophistica- 
tion, than those of the more research-oriented groups such as are found 
among the ABA membership. 

These results are supported by the findings on the related question 
"To what extent do current test reviews meet your needs?". The response 
distributions for that question are shown in Table 4. As Table 4 indi- 
cates, 51.6% of the total group indicated that current test v>:vlews were 
adequate, with more than twice as many of the remaining 50% indicating 
that test reviews were "too technical" as opposed to "too practical". 

Among the 11 organisations sampled, statistically significant differences 
were again found in response to this question. While only 15 . 91 of ABA 




Table 3 

Percentage distributions for Question 2: How do you feel about the nature of current test reviews? 
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Division 14 members Indicated that test reviews were "too technical", 

57.1% of ASCA members chose that response. Over 20% of AERA Division D, 

APA Division 14, and Division 15 members chose the "too practical" alterna- 
tive, while only 4.1% of ASCA and 6.5% of AMEG members agreed that test 
reviews are "too practical". 



Insert Table 4 about here 



Interestingly enough, 26.7% of the journal editors Indicated that test 
reviews were "too technical", while 60% supported the statement that test 
reviews were adequately meeting the needs In the field. On this question*’ 
journal editors tended to agree more with the members of the APA than with 
members of the applied groups represented by APGA membership. 

Depth, availability, and frequency of test reviews 

Test users were asked "Do you feel that current methods of dissemination 
of test reviews are adequately meeting the needs In the field In terms of the 
depth, availability and frequency of test reviews?" Users were requested to 
respond "yes" or "no" to each of these three possible problems. Percentage 
distributions for the percent responding "yes" to each of these three alterna- 
tives are shown In Table 5. 



Insert Table 5 about here 

Depth . Among the total group, 46.8% Indicated that current test reviews 
meet existing needs In terms of "depth". Analysis of the group differences of re- 
sponse to this question Indicated no statistically significant differences In re- 
sponse among the members of the 11 organizations sampled. Percentages varied from 
34.7% for APA, Division 12 to 55% for NVGA members. In general, then, It 
would appear that only about half of the respondents to this question Indicated 
that the "depth" of current test reviews Is appropriate. 

Availability . The percentage responding "yes" to the question vt "availa- 
bility of test reviews" was only 32% for the total group. The sub-group per- 
centages ranged from 22.7% for APA Division 16 members to 43.6% for APA Divi- 
sion 15 members. However, no significant differences occurred among the sub- 
groups. The data Indicate, however, that two out of three test users are not 
satisfied with the availability of test reviews. 

Frequency . The picture appears to be even more bleak for the question of 
"frequency of test reviews". Only 22.9% of the total group Indicated that 
test reviews are frequent enough In terms of £he needs of the field. Responses 
for the subgroups varied from 10.7% for APA Division 16 (9 our of 10 of respond- 
ents to this question said that test reviews are not frequent enough) to 
32.2% for members of SPTPA. Differences among the subgroups were statistically 
significant. Indicating different needs among subgroup members concerning 
timeliness of test reviews. 

These data point up three Important problems In test reviewing. First, 
almost eight out of ten test users feel that test reviews are not frequent 
enough. Seven out of ten feel that test reviews are not readily available, 
and slightly over half of the test reviewing audience feels that test reviews 
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Table 4 

Percentage distributions for Question 4: 



To what 


extent do current 


test reviews 


meet your needs? 




Group 


N 


too 

technical 


adequate 


too 

practical 


Total. Group 


1011 


34.7 


51.6 


. 13.6 


AERA, Dlv. D 


93 


20.4 


57.0 


22.6 


ARCA 


97 


49.5 


42.3 


8.2 


ASCA 


98 


57.1 


38.8 


4.1 


AMEG 


123 


39.0 


54.5 


6.5 


NVGA 


120 


41.7 


50.8 


7.5 


SPTPA 


93 


31.2 


49.5 


19.4 


APA, Dlv. 12 


67 


25.4 


56.7 


17.9 


APA, Dlv. 14 


82 


15.9 


63.4 


20.7 


APA, Dlv. 15 


72 


26.4 


51.4 


22.2 


APA, Dlv. 16 


73 


38.4 


43.8 


17.8 


APA, Dlv. 17 


93 


25.8 


61.3 


12.9 


Journal Editors 


15 


26.7 


60.0 


13.3 



x 2 (20 df) - 86.0 p < .01 
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Table 5 

Number of individuals and percent! choosing "yes" for the sub— parts of Question 
®o yot feel that current methods of dissemination of test reviews a . . 
are adequately meeting the needs in the field in terms of... 
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are not adequate in term of depth. These results are similar to those iden- 
tified in the early meetings of IACTR with organizational representatives, 
and were the kind of information which lead to the development of several 
prospective solutions to some of these problems. These solutions formed the 
second half of the survey. 



Prospects 



General Direction of New Programs 

Test users and journal editors were asked "Should new test reviewing 
efforts concentrate on "more test reviews", "different types of test 
reviews", and "bibliographies for tests". Respondents were asked to reply 
categorically "yes' or 'no" for each of the three alternatives, then to 
indicate which of the three was most important. Percentage distributions 
of responses to these questions are shown in Table 6. 

Insert Table 6 about here 



As Table 6 indicates, 70. IX of the 975 test users responding to this 
question indicated that new test reviewing efforts should be concerned vrith 
"more test reviews". Among the responding organizations, the percentage 
responding "yes" to the question of "more test reviews" varied from 62. 5% 
for members of ASCA to 79.3% for members of Division D of /.ERA. The chi-square 
value of 13.7 for group differences in the responses to this question was 
not statistically significant at the ,05 level. Hence the total group per- 
centage of 70% would be representative of the responses for each of the sub- 
groups. Journal editors agreed with test users on this question. A total 
of 77.8% of the journal editors responding to this question indicated that 
"more test reviews" is an appropriate direction for future efforts in the 
test reviewing field. 

Concerning the question of "different types of test reviews", 89.9% of 
the total group endorsed this as an appropriate direction for future efforts. 
Among the sampled groups, 79.3% of APA Division 14 members indicated that 
"different types' of test reviews were appropriate, while 94.6% of APA 
Division 17 members expressed a preference for this alternative. The differ- 
ences among organizations in response to this question were statistically 
significant at the 5% level. This, of course, supports the previously ob- 
served differences among responding subgroups in terms of their dissatisfac- 
tion with the nature of current test reviews. However, the preferences 
expressed in responses to thi6 question for new kinds of test reviews are 
even more marked than they were in responses to previous questions. Among 
journal editors, 37-5 % agreed that 'different types" of test reviews were 
appropriate. Again journal editors and test users vrere in general agree- 
ment as to the future direction of new test reviewing efforts. 

The question of 'bibliographies for tests" yielded a somewhat different 
picture. Uhile 63.3% of the total group indicated that new test reviewing 
efforts should include bibliographies, the percentages varied widely among 
the responding subgroups, and results were statistically significant. Only 
8 lightly more than half (53.0%) <jf the APA Division 16 members endorsed the 




Number of individuals and percent choosing "yes" for the sub-parts of Question 5: 

Should new test reviewing efforts concentrate on... 
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need Cor bibliographies, while 77. 2% of the ABBA Division D members supported 
that prospect. This would suggest that the more research-oriented groups, 
such as AERA, have a greater need Cor bibliographic material than the more 
applied groups such as Divisions 16 and IS oC AEa and NVGA members. Journal 
editors were In general agreement with the total group, as 62.5% oC the Journal 
editors endorsed the need Cor bibliographies. 

When the test users were asked to Indicate which of the three activities 
was the most important direction for new test reviewing efforts, the preference 
was clearly for "different types" of test reviews. Among the total group, 

57.9% indicated that "different types" of test reviews were the most important, 
while only 14% supported "bibliographies" and 28% supported "more test reviews". 
The .chi-square analysis of the response distributions for this comparative 
question indicated no statistically significant differences among the respond- 
ing subgroups. All groups agreed that "different types" of test reviews are 
the most important of the three options given. 

Type of reviews 

Anticipating the need for different types of test reviews, based on 
earlier discussions with representatives of the organizations Involved in 
IACTR, the questionnaire presented four different types of test reviews to 
both the Journal editor and the test user audience. The question in both 
surveys was worded as follows "For each type of test review listed below 
indicate whether or not you feel it is (or could be) an important kind of 
test review: (a) comprehensive and integrative reviews of the research 

on a te 3 t (b) comparative reviews of measuring instruments of the same type 
(e.g„, reviews comparing the Strong Vocational Interest Blank and the Kuder 
Occupational Interest Survey, or the Stanford-Binet and the Wechsler Adult 
Intelligence Scale); (c) topical reviews, e.g., reviews of the Measurement 
of intelligence 1 , reading, achievement, etc. (d) reviews of new tests as 
they appear, based primarily on the information in the manual." Respondents 
were directed to reply "yes" or "no" to each of these four alternatives. In 
addition, the survey questionnaires asked the following question "Of the 
four types of reviews listed above, which type do you feel is the most import- 
ant ?" This question was designed to obtain an indication of the relative 
Importance of the various ty-ies, in terms of setting priorities for the 
development of new test reviewing efforts. 

Table 7 shows the percentage responding "yes" to each of the four 
alternatives, and the percentage choosing each of the alternatives as the 
"most important" typo of review. 



Insert Table 7 about here 



Comprehensive and integrative reviews . Among the total group, 88.2% of 
the 1,055 respondents indicated that '‘comprehensive and integrative" reviews 
of the research on a test was an important kind of test review. Response 
percentages among the organizational subgroups varied from 73.87. for the 
members of NVGA to about 957. for members of Divisions 12 and 14 of AFA. 
Differences were statistically significant among the organizational sub- 
groups, reflecting the different needs of the groups, based probably on the 
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activities of the constituent members. Journal editors endorsed this alter* 
native at the rate of 95.2%, agreeing more with APA members than with members 
of the applied organizations. 

Comparative reviews . About 957. of the total group of test users and 
957. of the group of journal editors agreed that comparative reviews were an 
important kind of review. Analysis of the differences in percentages among 
the test user subgroups showed that only 917. of APA Division 15 members 
supported the need for comparative reviews while 98.57. of AMEG members 
desired comparative reviews of tests. Differences among organizational 
groups were not statistically significant. It is important to note that 
while over nine out of ten of the members of all groups studied folt that 
comparative reviews were important kinds of reviews, very few of the reviews 
now being published are of a comparative nature. 

Topical reviews . Support for the alternative of topical reviews was 
expressed by 87.6% of the test users and 90% of the joivnal editors. Among 
the organizational subgroups, only 76.7% of APA Division 12 members endorsed 
this alternative while 90.6% of APA Division 17 members supported the 
notion of topical reviews. However, differences among organizational sub- 
groups were not statistically significant. Agaitv the topical review which 
is rarely now available was endorsed by almost nine out of ten members of 
all groups studied. 

Hew tests . Test users and journal editors disagreed with each other 
on whether reviews of new tests were desirable. Of the total group of test 
users, only 58.9% of the respondents endorsed the need for reviews of new 
tests. Among journal editors 757. endorsed this alternative. This difference 
suggests that journal editors perhaps are not fully aware of the desires of 
test users, and are producing test reviews of new tests because of their in- 
accurate perception of the needs of test users. 

Among the test users subgroups, only 50% of ARCA members were Interested 
in the reviewing of new tests, while 63% of ASCA members were Interested in this 
type of test review. However, the differences among test user subgroups were 
not statistically significant. 

Comparison of types of test reviews . When test users were asked to choose 
among the four types of test reviews in terms of the one which is "most impor- 
tant", the total group's results were equally divided between two types of 
reviews that are frequently not currently available. About 38% of the total 
group of test useivj indicated that the "comprehensive and integrative" reviews 
were most important, while an additional 38% indicated that "comparative 
reviews" were most important. It is interesting to note that for the type 
of test reviews currently most prevalent in reviewing Journals, only 6.6% of 
the total group indicated that the reviewing of "new tests" was the most 
important of the four types of reviews; ninety-three out of 100 respon- 
dents indicated that other types of reviews were more important. 

Interestingly enough, similar results were found for journal editors. 

Only 4.57. indicated that reviews of "new tests" were most important (contrary 
to the current practice in Journal test reviews), while half of the journal 
editors felt that comprehensive and integrative reviews of the research on a 
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test were the most important of the four types given. The journal editors 
disagreed, however, with the test users in terms of the desirability of 
topical reviews vs. comparative reviews. About 32% of journal editors 
supported topical reviews as most important while only almost half that 
proportion (16.7%) of test users endorsed topical reviews as most important 
of the four alternatives. Test users felt comparative reviews to more 
important (38.4%) while Journal editors felt these to be less important 
(13.6%). These findings can possibly reflect the differential needs of the 
average test user and the perceptions of those needs by Journsl editors in 
terms of the resesrch vs* practical nature of test reviews. 

Analysis of the subgroup differences in responses to the comparative 
question of type of test reviews indicated a highly statistically significant 
difference in responses among these subgroups. For five of the 11 subgroups, 
comprehensive and integrative test reviews were the most important of the 
four types listed. These groups were the SPTPA and the following four divi- 
sions of APA: Divisions 12, 14, 15 and 17. For these groups about 50% of 

the choices were for comprehensive and Integrative types of test reviews. 
Comprehensive and integrative reviews were also chosen most frequently by 
members of AERA Division D; however, the difference between the percentage 
choosing that response (37.8) and the percentage choosing comparative 
reviews (33.3%) wss not as great. 

Four of the 11 groups clearly chose comparative reviews as mcs t important 
of the four types. Percentages for these four groups ranged from 42.9% (APA 
Division 17) to 52.8% (ASCA members). For none of the groups was topical 
reviews or reviews of new tests the most important of the four types of 
reviews. The percent choosing new tests as the most important varied from 
a low of 2.2% for APA Division 16 to a high of 11.8% for members of AMEG. 

These data indicate, in general, that different test reviewing populations 
require different kinds of reviews to meet their needs. 

Dissemination 



The yearly discussions of IACTR and the preliminary Division 5 subcocmittee 
discussions also were concerned with new methods for dissemination of test 
reviews. Accordingly, both the journal editors survey and the test user survey 
included the following question: "Should test reviewing efforts expand in 

the direction of q) A separate periodical publication devoted exclusively to 
a variety of types of test reviews for all kinds of tests; b) A series of 
separate publications by broad subject matter headings, e.g. , tests of intelli- 
gence, measures of personality, Elementary School tests, Secondary School 
tests, etc. c) Separates for individual tests available from a central source 
by mail; and d) More reviews in existing journals." Both Journal editors and 
test users were asked to respond "yes" or "no" to each of these four alterna- 
tives. In addition the test user groups were ssked to indicate "Of the four 
possibilities, which is the most important?" 

Periodical publication . Table 8 shows the percentage distribution of the 
responses to this series of questions. For the total group of test users, 

58.9% endorsed the possibility of a separate periodical publication, while 
47% of the Journal editors supported that means of dissemination. Among 
the 11 test user groups sampled, responses ranged from 45.3% "yes" for ABA, 
Division 14 members to 67.5% for members of AMEG. Differences among 
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the groups were statistically significant. In general, the Idea of a separate 
periodical publication was supported more by groups which were not members of 
APA, with the exception of SPTPA. 



Insert Table 8 about here 



Separate subject matter publications . On the question of separate subject 
matter publications, 69% of the total group supported that as an alternative. 
Por this question, only 47% of Journal editors supported the Idea of a separate 
subject matter publication. Among the 11 responding groups, percentage endors- 
ing this alternative ranged from 56.87. for members of Division 15 of APA to 
a high of 88.3% for ASCA members, with differences statistically significant 
at the .01 level. It is interesting to note that the percentage responding 
"yes" for Journal editors was substantially below the percentage responding 
"yes" for almost all of the test user groups. 

Separates available by call . The difference in opinion between Journal 
editors and test users was even more pronounced on the question of the availa- 
bility of separate test reviews for Individual tests through a "mail order 
house". Only 257. of Journal editors endorsed this alternative, while over 
twice as many (57 . 77.) of the total group of test users endorsed this possi- 
bility. While subgroups of the test users varied from 48.97. to 68.9% "yes", 
the differences among test user groups were not statistically significant. 

More reviews In existing journals . On the question of more reviews in 
existing Journals, the Journal editors and test users again disagreed some- 
what. Among the Journal editors, 72.27. supported this possibility, while 
only 58.6% of test users supported the dissemination of more test reviews 
in existing Journals. The test user sub-groups responded significantly 
differently on this question. Only 457. of members of SPTPA supported this 
alternative dissemination mechanism, compared to a high of 69.3% of the ' 
members of APA Division 15. 

Comparison of dissemination approaches. In comparing the four alterna- 
tive methods of dissemination of test reviews, statistically significant 
differences were found among the 11 test user groups. The members of ASCA 
clearly chose "separate subject matter publications" as the most important 
(52.77.) as did members of NVGA. Members of most other organizations also 
chose this approach as most important, although the result was not as clear. 
SPTPA members were equally distributed between "separate periodical publica- 
tion" and "separate subject matter publications" as their first choice. 

AMEG members preferred the "separate periodical publication" to "separate 
subject matter, publications" by a small margin. While none of the groups gave 
a majority preference to the possibility of more reviews in existing Journals, 
the percentages responding to this as the most appropriate alternative varied 
from 5.47. for ASCA members to 24.77. for APA Division 15 members. These results 
tend to suggest, however, that different dissemination mechanisms might be 
appropriate for different subgroups of the teat user population. 

Prospects for subscribers . In an attempt to obtain a more definite 
statement of the "market" possibilities for some specific new methods of 
dissemination, the following question was asked of test users: "Which of 

the following new methods of dissemination of test reviews would you patro- 
nize? a) a reprint service from which you could purchase all reviews done 




Number of individuals and percent choosing "yes" for the sub-parts of Question 6: Should 

test reviewing efforts expand in the direction of... 
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teat were the most important of the four types given* The journal editors 
disagreed^ however, with the test users in terms of the desirability of 
topical reviews vs* comparative reviews* About 32% of Journal editors 
supported topical reviews as most important while only almost half that 
proportion (16*7%) of test users endorsed topical reviews as most important 
of the four alternatives* Test users felt comparative reviews to more 
important (38,4%) while journal editors felt these to be less important 
(13*6%)* These Tlndings can possibly reflect the differential needs of the 
average test user and the perceptions of those needs by Journal editors in 
terms of the research vs. practical nature of test reviews. 

Analysis of the subgroup differences in responses to the comparative 
question of type of test reviews indicated a highly statistically significant 
difference in responses among these subgroups. For five of the 11 subgroups, 
comprehensive and Integrative test reviews were the most important of the 
four types listed* These groups were the SPTBA and the following four divi- 
sions of ABA: Divisions 12, 14, 15 and 17* For these groups about 50% of 

the choices were for comprehensive and integrative types of test reviews* 
Comprehensive and integrative reviews were also chosen most frequently by 
members of AERA Division D; however, the difference between the percentage 
choosing that response (37*8) and the percentage choosing comparative 
reviews (33.3%) was not as great. 

Four of the 11 groups clearly chose comparative reviews as mce t Important 
of the (our types. Percentages for these four groups ranged from 42.9% (ABA 
Division 17) to 52.8% (ASCA members)* For none of the groups was topical 
reviews or reviews of new tests the most Important of the four types of 
reviews. The percent choosing new tests cs the most Important varied from 
a low of 2.2% for ABA Division 16 to a high of 11.8% for members of AMEC* 

These data Indicate, in general, that different test reviewing populations 
require different kinds of reviews to meet tbelr needs. 

Dissemination 



The yearly discussions of IACTR and the preliminary Division 5 subcommittee 
discussions also were concerned with new methods for dissemination of test 
reviews* Accordingly, both the Journal editors survey and the test user survey 
Included the following question: "Should test reviewing efforts expand in 

the direction of q) A separate periodical publication devoted exclusively to 
a variety of types of test reviews for ail kinds of tests; b) A series of 
separate publications by broad subject matter headings, e.g., rests of intelli- 
gence, measures of personality, Elementary School tests, Secondary School 
testa, etc. c) Separates for individual tests available iVou a central source 
by mall; and d) More reviews in existing journals." Both Journal editors and 
test users were asked to respond "yes" or "no" to each of these four alterna- 
tives. In addition the test user groups were asked to indicate "Of the four 
possibilities, which is the most important?" 

Periodical publication . Table 8 shows the percentage distribution of the 
responses to this series or questions. For the total group of test users, 

58.9% endorsed the possibility of a separate periodical publication, while 
47% of the Journal editors supported that means of dissemination. Among 
the 11 test user groups sampled, responses ranged from 45.3% "yes" for ABA 
Division 14 members to 67.5% for members of AHEO. Differences among 
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the groups were statistically significant* In general, the idea of a separate 
periodical publication was supported more by groups which were not members of 
ABA, with the exceptioa of SPTPA. 



Insert Table 8 about here 



Separate subject matter publications * On the question of separate subject 
matter publications, 69% of the total group supported that as an alternative* 
For this question, only 47% of Journal editors supported the idea of a separate 
subject matter publication. Among the 11 responding groups, percentage endors- 
ing this alternative ranged from 56*8% for members of Division 15 of APA to 
a high of 88.3% for ASCA members, with differences statistically significant 
at the .01 level. It la interesting to note that the percentage responding 
"yes" for Journal editors was substantially below the percentage responding 
"yes" for almost all of the test user groups. 

Separates available by mail . The difference in opinion between Journal 
editors and test users was even more pronounced on the question of the availa- 
bility of separc'e test reviews for Individual tests through a "mall order 
house". Only 25% of journal editors endorssd this alternative, while over 
twice as many (57.7%) of the total group of test users endorsed this possi- 
bility. While subgroups of the test users varied from 48.9% to 68.9% "yes", 
the differences among test user groups were not statistically significant. 

More reviews in existing journals . On the question of more reviews in 
existing Journals, the Journal editors and test users again disagreed some- 
what. Among the Journal editors, 72*2% supported this possibility, while 
only 58*6% of test users supported the dissemination of more test reviews 
in existing Journals. The test user sub-groups responded significantly 
differently on this question. Only 45% of members of SPTPA supported this 
alternative dissemination mechanism, compared to a high of 69.3% of the' 
members of ABA Division 15. 

Comparison of dissemination approaches. In comparing the four alterna- 
tive methods of dissemination of test reviews, statistically significant 
differences were found among the 11 test user groups. The members of ASCA 
clearly chose "separate subject matter publications" as the most important 
(52.7%) as did members of HVGA* Members of most other organisations also 
chose this approach as most important, although the result was not as clear. 
SFIPA members were equally distributed between "separate periodical publica- 
tion" and "separate subject matter publications" as their first choice. 

AMEO members preferred the "separate periodical publication" to "separate 
subject matter publications" by a small margin. While none of the groups gave 
a majority preference to the possibility of more reviews in existing Journals, 
the percentages responding to this as the most appropriate alternative varied 
from 5.4% for ASCA members to 24.7% for ABA Division 15 members. These results 
tend to suggest, however, that different dissemination mechanisms might be 
appropriate for different subgroups of the test user population. 

Prospects for subscribers . In an attempt to obtain a more definite 
statement of the "market^ 1 possibilities for some specific new methods of 
dissemination, the following question was asked of test users] "Which of 
the following new methods of dissemination of test reviews would you patro- 
nise? a) a reprint service from which you could purchase all reviews done 
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on a teat; b) a review subscription service which would provide you with 
reviews in given subject matter areas (e.g., intelligence, personality, 
reading) as they were available; c) a quarterly Journal which would publish 
reviews of all tests and a variety of kinds of reviews (comparative, inte- 
grative, new testi'); d) hard cover collections of reviews on a subject 
matter area (e.g,, reviews of reading tests, intelligence tests) published 
at 5-year Intervals," The test users were asked to respond "yea" or "no" to 
each of those four alternatives. 

The response distributions for the total group and the 11 subgroups are 
shown in Table 9, Among the total group 63,87, indicated that they would 
patronize a "reprint service" providing reviews of specific tests by name. 

Among the 11 organizational sub-groups, percentage responding "yes" varied 
from a low of 52,1% for members of Division 15 of APA to a high of 77,4% for 
AMEG members. Differences among the subgroups were statistically significant. 
The results show, however, that at least 50% of the members of all groups 
sampled indicated that they would patronize a reprint review service. General- 
ization of these results to the total organizational membership of the groups 
sampled indicates a very large possible source of users of such a reprint 
service. 



Insert Table 9 about here 



The possibility of a review subscription service was supported by 67.8% 
of the total group. While subgroup differences varied from 55.7% for APA 
Division 15 members to 76.9% for ASCA members, the differences among sub- 
groups were not statistically significant. However, about two out of every 
three members of the organizations studied indicated their probable patro- 
nage of a review subscription service. 

On the question of a quarterly journal, again almost two out of every 
three (65.8%) of the total group indicated that they would support such a 
service. The percentage responding "yes" varied from 46*5% for APA Division 
12 members to 75.6% for NVGA members. The differences between the 11 sub- 
groups were statistically significant, indicating a differential interest 
in a new Journal of test reviews on the part of the members of various 
organizations. However, at least about one out of two of all respondents 
supported the creation of a quarterly Journal of test reviews, with about 
three out of four responding affirmatively among the less highly academically 
orlentt-d groups. 

The possibility of a series of hard cover subject matter collections 
(similar to the kinds of review collections currently being published by the 
Mental Measurements Yearbooks, e.g., Personality Tests and Reviews , Read ins 
Tests and Reviews ) was received favorably by only 35.6% of the total group. 

The organizational subgroups responded significantly different on i.<*s question, 
however. Only 18,67* of the ASCA members supported the possibility of a hard 
cover subject matter collection, while proportionately three times as many of 
the APA Division 15 members (58.6%) supported the possibility of a hard cover 
subject matter publication. The differences in endorsement percentages may 
relate to the relatively high cost of subject matter publications, and the 
availability of such funds among specific professional subgroups. These data 
might reflect the fact that the groups responsible for the day-to-day operations 



Table 9 

■ of Individuals and percent choosing "yes” for the sub-parts of 
scion 8: Which of the following new methods of dissemination 

of test reviews would you patronize? 




m 

co 



R 

O 



00 

vO 



ON 

rH 

O 



00 

vO 



8 



CO 

m 

vO 



R 

On 



O 



3 

$ 



- 21 - 



sO 

O 

m 



c> 



co 



ON 



NO 

oo 



£ 



CO 

a 

CO 



vf> 

9 

rH 

CM 



2 S 



m 



m 

co 



55 

O s 



n* rH 

CO *0 



ON 

nO 



s 



nO 

a 



r 



m 

co 



CM 



o 

co 



cm 

ON 



CO 

• 

On 

so 



8 



CM 

a 

n 

vD 



CM 

0% 



CO 

r* 



vO 

a 

m 



co 

a 

ON 

m 



uo 



CM 



ON rH 



CM 



* 3 £ £ R 



3 a 8 



rH 



8 3 R 3 



m on 



8 



$ 



00 



rH 

nO 



rH 

On 



NO 



CM 

a 

in 



vO 

a 

ON 

vO 



On 

a 

o 

vO 



© m 

«. a 

O Ch 

N \[> 



£ ^ m 

ON M rH 



CM O 

on rs. 



3 3 



m 

m 



'O rH 

a a 

I s * co 

SO 



On 

m 

nO 



CM 

sO 



CO 

00 



On 

m 

NO 



in rH 

m 



R R 



CM 

a 

m 

sO 



CM 

On 



CM 

m 



ON 

NO 



8 



Sf jH © in '■H rH O in rH ON rH 
ONONO>fHrHONf^CO^N0ON 
rH rH 



6 

a 



cm ^ in no 

rH rH rH rH rH 

a • • a a 

^ ► ► ► ► 

s s s s s 



H H I I 4 i i i i 




Chi-square 18.3 15.8 37.7 47.8 



of service agencies (e.g., ASCA, NVGA, APA Division 17) have less funding 
available for such purchases than those Involved In research (e.g., APA 
Division IS and AdRA Division D). Alternatively, these results nay reflect 
the perceived archival and research function of the "collections" of reviews 
and/or the delay in publication specified by the "5-year interval" clause 
in the question. 

Taken together, however, the three new dissemination mechanisms suggested 
in the survey**" a reprint service, a review subscription service, and a 
quarterly journal--were favorably received by about two out of every three of 
the individuals included in this survey, while the hard cover subject matter 
collection was endorsed by only one out of three people surveyed. This 
"desire to patronise" the new approaches to dissemination of test reviews 
implies a very large group of potential subscribers for any or all of the 
proposed publication services. 

Who should review tests? 



One of the concerns of the XACTR representatives was that of determin- 
ing whether changes were desirable in the test reviewing process as It is 
currently carried out. This included a concern with developing new sources 
of reviewers, or new ways of developing test reviews. Accordingly, the 
following question was asked of test users: "Who should review tests 

(assuming that each choice is among "experts" in the field)? a) one 
expert only; b) more than one expert, but independently of each other; 
c) more than one expert in consultation with each other; d) a penal of 
experts chosen specifically for tests of a given type; e) a panel of 
experts who continuously review all tests," Test users were asked to reply 
"yes" or "no" to each of these five alternatives, In addition, the following 
question was asked "Which is the most appropriate type of reviewing process?", 
in order to obtain a comparative statement concerning the five possibilities 
previously presented. 

The results of the responses to this series of questions are shown in 
Table 10, In the total group only 5% indicated that "only one expert" should 
review tests. Differences among the organisational subgroups were statistically 
significant however, with less than 1% of the NVGA members supporting this 
alternative, but 12.5X of the APA Division 14 members indicating that only one 
expert should review teste. Of the total group, 78,4% supported the second 
alternative-- "more than one expert, but Independently of each othar". Again, 
differences among subgroups were statistically significant with only 67,6% of 
NVGA members supporting this alternative while 88.5% of APA Division 17 
members supported this method of test reviewing, the most cocmon one in use 
now. 



Insert table 10 about here 



The possibility of test rsvlsws being done by more than one expert In 
consultation with each other wee supported by only 43,9% of the total group. 
Largs cod statistically significant between group differences were observed 
on this question. Only 23.3% of APA Division IS memusrs supported the use of 
"experts in consultation", whereas proportionally twice as many of the ASCA 
members (59.8%) supported this possibility. On the question of e "panel of 



Humber of individuals and percent choosing "yes'* for the sub-parts of Question 10: 
Who should review tests (assuming that each choice is among "experts" In the field)? 
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Chi-square 20.9 43.4 32.8 7.3 28.5 73.7 



reviewers", e method almost never used in current test reviewing procedures, 
a surprising 79*4% of the total group endorsed this possibility* It should 
be noted that this percentage is about the same as the percentage endorsing 
the current method of having more than one Independent test review* Ga 
this question there were no statistically significant differences among 
groups, indicating that all of the groups sampled were in agreement on this 
question* When the question was posed as "a panel of test reviewers for all 
tests" (as compared to a panel for specific teste) the percentage endorsing 
this alternative dropped to 41*6% for the total group* The responses to this 
question were more similar to the responses to the question of "more than one 
expert in consultation with each other", since the two methods are fairly 
similar* Again, significant differences were found between the groups with 
24. IX of A FA Division 14 members responding positively while over half 
(53*C%) of NVGA membere supported this alternative* 

On a comparative basis, within the total group the second and fourth 
alternatives received almost equal percentages of first choice* About 
35X of the total group supported the current method of "more than one expert, 
but independently" as the most appropriate type of reviewing process, while 
39*9% (or slightly more) supported the possibility of "a panel of test 
reviewers for specific teste"* As might be expected from the previous respon- 
ses, statistically significant between group differences were observed In 
percentage responding to eaoh of the five alternatives as "the most appropriate 
type of test reviewing process"* Pour of the subgroups (ABRA Division D, 
and Divisions 12, 14 and 15 of APA), chose the current method of reviewing 
as the most appropriate method* However, in each case the "panel for 
spaolflo teats" alternative was second highest, aod for APA Division 12 
members the percentages for the two methods were equal* 

For the remaining seven groups, the most appropriate method of test 
reviewing wee indicated to be'k panel for tpeolflo tests"* However, A8CA 
members divided their choices almost equally among the last four alterna- 
tives as compared to members of APA Divisions 14 end 15 which were heavily 
in favor of the second and fourth alternatives* 

Zt is interesting to note that the percentages choosing the eltenwtlve 
"more than one expert in consultation with each other" varied from a low of 
2*3X for APA Division 14 to 20% for ASCA members* In terms of "a panel for 
all testa", 8*1% of APA Division 14 members felt this the most appropriate 
method while proportionately three tiiuss as many of tbs ASCA members (24*2%) 
felt that to be the most appropriate method of reviewing tests* The alter- 
nstlve of "one expert only" was uniformly selected to be the least appro- 
priate test reviewing process with percentages of from 0% (APA Div. 15 members) 
to only 4*7% (APA Division 14) choosing that as tbs most appropriate test 
reviewing process* These data seem to suggest, therefore, thet tests should 
be reviewed by multiple individuals, but the exact mechanism for implementa- 
tion appears to dlffar from organisation to organisation* 

Other results of the surveys 



Test users etrvay 

Obtain ins Information on tests * Test users wars also asked "Xo choosing 
a test tor a specific purpose how do you currently obtain informatlonf"* The 
results of the responses to this question are shown in Table 11. Among the 




total group 80*4% Indicated that they "asked colleagues" for information on a 
test. The subgroup differences were not significant on this question. Only 
39*4% of the total group indicated they "consult test experts". Subgroup 
response differences were statistically significant. About 79% of the 
vejpondents indicated that they "search journals" to obtain information on 
tests, with the differences among subgroups significant only at the 10% level. 
On the question of "consulting the Mental Measurements Yearbook s , 90.7* 
the total group indicated that they do use those publications as a source of 
information. However, for this question, significant differences were found 
among the organisational subgroups. Only 77.9% of the ASGA members Indicated 
"consulting Mental Measurements Yearbooks" as a source of information, while 
98.4% of the AMEO group (or virtually all of the AMSG members) use the MKYs 
as a source of information on tests. Among the total group, 21.8% indicated 
that they use "other" sources of information on tests. An analysis of the 
"open-end" responses to this question indicated that the majority of test 
users do one of the following to obtain information on tests (in decreasing 
order of appearance): 1) read the technical manual, 2) write to test 

publishers for information, 3) examine a specimen set of the test, 4) talk 
to test salesmen, or 5) experiment with the test in trial administrations 
or in tbelr own research. 



Insert Table 11 about here 

When test users were asked to indicate which of the five sources of 
information were their "primary source" of information on tests, statistically 
significant differences appeared among the 11 subgroups. In the total group, 
44.6% indicated that the Mental teaaurements Yearbooks was their primary source 
of information on tests. Similar results were obtained for nine of the eleven 
subgroups, with percentages choosing the Mental Measurements Yearbooks as the 
primary source of intonation ranging from 27.3% for ABA Division 12 members 
to 57.1% for ABRA Division D members. Among the two clinical subgroups, (49A 
Division 12 and 8PTFA) the Mental Me asu rements Yearbooks did not appear as the 
primary sourca of intonation on tests. For 8FTPA, the Mental M easurements 
Yearbooks and "colleagues" obteinsd equal percentages of 34.1% as the primary 
source of information. Among the members of AFA Division 12, the primary 
source of information was "colleagues" with 36.4%, while the Mental M easure- 
ments Yearbooks obtained only 27.3%. Apparently very little use is made of 
test experts as primary sources of information, the percentages varying from 
0% for three of the 11 groupe to 10.8% for AMEO members. 

The Mental Measurements Yearbooks . Two additional questions were asked 
of teat users concerning the Mental Measurements Yearbooks , since it tea a 
primary source of information oi tests. Table 12 shows a distribution of 
responses to determine the accessibility of the Mental Measurement Yearbooks 
to test users. As few as 1.8% of ABRA Division D members said that they had 
access to "none" of the Mental Measurements Yearbooks, while 26*2% of ASCA 
members did not have access to anv ol! the Mental Measurements Yearbooks * Table 
12 also shows the distribution of psrcentsgss responding positively to the 
question of access to eaoh of the volumes of M(Ys,irlth uniformly high percen- 
tages of response to the stoat recent volume. 

Insert Table ia about here 
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Table 12 

Percent responding to the sub-parts of Question 11: 

To which volumes of the Mental Measurements Yearbooks 
do you have ready access (e.g., in a nearby library)? 



Group 


N 


None 


Volume 6 


Volume 5 


Volume 4 


Volumes 
3, 2. or 1 


Total Group 


1147 


8.7 


70.5 


71.1 


59.5 


46.8 


ABRA, Dlv. D 


112 


1.8 


85.7 


89.3 


78.6 


67.9 


ARCA 


103 


6.8 


68.0 


79.6 


64.1 


49.5 


ASCA 


103 


26.2 


52.4 


46.6 


34.0 


28.2 


AMEG 


132 


6.1 


70.0 


76.5 


62.9 


45.4 


NVGA 


130 


6.2 


60.8 


60.0 


46.9 


39.2 


SPTPA 


107 


11.2 


66.4 


64.5 


53.3 


37.4 


APA, Dlv. 12 


85 


16.5 


60.0 


60.0 


45.9 


37.6 


APA, Dlv. 14 


98 


2.0 


70.4 


76.5 


65.3 


49.0 


APA, Dlv. 15 


90 


5.6 


80.0 


77.8 


70.0 


64.4 


APA, Dlv. 16 


87 


11.5 


67.8 


64.4 


54.0 


36.8 


APA, Dlv. 17 


100 


5.0 


85.0 


85.0 


79.0 


60.0 



O 

ERLC 



Table 13 shows Che percentage distribution of reeponsee Co the question 
"How valuable do you consider the comprehensive test bibliographies that 
appear in the Mantel Measurements Yearbooks ? 11 * The response alternatives 
were as follows: Extremely valuable-- they should definitely be continued; 

valuable*- they should be continued if possible; Indifferent; not valuable— 
the space could be better spent on more or different kinds of reviews and; 
worthless— they take up too much space end tbqr raise the price of the Mental 
Measurements Yearbooks unnecessarily*" As Table 13 shows, only IX of the 
respondents in the total group indicated that the bibliographies are "worth- 
less"* Only 5*5% indicated that they were "not valuable". Over 75X of the 
respondents indicated that they were "valuable" or "extremely valuable". 

Insert Table 13 about here 



As Table 13 also shows, significant differences were obtained in the 
distributions, of responses to this question among the 11 subgroups* There 
was a tendency for members of APA Division 12 to view tbs Mental Measurements 
Yearbooks bibliographies as either "indifferent" or 'hot valuable", while 
members of ABRA Division D were least likely to respond to those to alterna- 
tives* The percentage of respondents indicating that the bibliographies in 
the Mental Measurements Yearbooks were "extremely valuable" varied from 
14. OX for ASCA members to 36*5X lor ABRA Division D members* These percen- 
tages undoubtedly reflect the differential activities of these groups as they 
relate to the use of bibliographic materials. 

Journal Editors Survey 

Several additional questions were asked of Journal editors, using two 
forma of the Journal Editors Survey Questionnaire* One form of the questionnaire 
was designed for use with those Journals that did publish test reviews; the 
other was designed for those Journals which do not review tests* In addition, 
several questions were common between the two forms* 

Table 14 shows the percent of Journal editors responding positively to 
each of the queatlons that cere common to both forms of the Journal Bdltors 
Survey Questionnaire* As Table 14 indicates, approximately 30X of the 
Journal editors considered test reviewing to be a desirable function of the 
Journal* Thirty percent also actively solicited test reviews for the Journal 
or considered publication of aubmltted test reviews which had not previously 
been solicited* When Journal editors were asked whether they felt it 
desirable that there be at least two reviews of a test in the some Journal, 

63*2X of the 19 Journal editors endorsed that alternative* Of the 19 Journal 
editors responding to the question in terms of whether it was desirable to 
have two reviews of tbs test appear in different Journals, 64.2% responded 
affirmatively* 
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One question in the Journal editors survey was concerned with the general 
question of whether current methods of test reviewing were meeting the needs 
of the test reviewing field* Only 10% of the 20 Journal editors responding to 
that question felt that this was true* 
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Table 13 

Percentage response distributions for Question 12*. 

How valuable do you consider the comprehensive test bibliographies 
that appear in the Mental Measurements Yearbooks ? 



Group 


N 


extremely 

valuable 


valuable 


indifferent 


not 

valuable 


worthless 


Total Group 


1062 


24.2 


51.8 


17.5 


5.5 


1.0 


AERA, Div. D 


104 


38.5 


51.0 


5.8 


3.8 


0.0 


A RCA 


99 


22.2 


54.5 


17.2 


6.1 


0.0 


ASCA 


93 


14.0 


59.1 


20.4 


4.3 


2.2 


AMEG 


125 


24.8 


57.6 


14.4 


3.2 


0.0 


NVGA 


123 


22.8 


48.0 


23.6 


5.7 


0.0 


SPTPA 


96 


27.1 


43.8 


24.0 


4.2 


1.0 


APA, Div. 12 


: 78 


24.4 


43.6 


17.9 


11.5 


2.6 


A PA, Div. 14 


87 


17.2 


63.2 


14.9 


3.4 


1.1 


APA, Div. 15 


81 


30.9 


43.2 


21.0 


3.7 


1.2 


APA, Div. 16 


78 


20.5 


47.4 


19.2 


9.0 


3.8 


APA, Div. 17 


98 


22.4 


55.1 


15.3 


7.0 


0.0 



X 2 (40 df) «» 65.6 p <.01 
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Table 14 

Responses of Journsl editors to additional common questions 



HI WH 

*'o you consider test reviewing to be a desirable 

function of your Journal?, 21 38.1 

Do you actively solicit test reviews for your Journal?. . .22 31.8 

Do you consider publication of submitted test reviews 

which have not been solicited ...20 30.0 

Do you feel that It Is desirable to have at least 
two reviews of a test appear In.,, 

1. the same journal..,. 19 63.2 

2. different journals 19 84.2 

Do you feel that current methods of dissemination of 
test reviews (l.e,, the Mental Measurements 
Yearbook snd two or three journals that publish 
reviews regularly) are adequately meeting the 
needs In the teat reviewing field? 20 10.0 



This should bo considered In contrast to the 38% which considered test review- 
ing to be a desirable function of their journal. In other words, while the 
journal editors felt that the needs were not being met In the field, they 
apparently were unwilling or unable to consider test reviewing as an Important 
function of the journal. 

Table 15 shows the responses of those journals which did review tests to 
the questions specifically asked of those Journals. The first two questions 
were concerned with possibility of IACTR obtaining copies of previous and 
future reviews that had been published In journals for distribution to test 
users either by free distribution or by purchase of test users. Eight journals 
replied to these questions. In response to the question of whether IACTR could 
reprint existing test reviews, half of the journals responded positively, half 
responded lndetermlnantly, and none responded negatively. When the question 
was posed In terms of whether IACTR could sell reprints of past reviews to 
test users, one journal said "no", two said "yes" and three replied lndeterml- 
nantly. The responses to the second question concerning future test reviews 
were Identical to those of the first question. 



Insert Table 15 about here 



Table 15 shows that four of the six journals replying to the question 
concerning the Influence of test reviews on selection decisions made by the 
readers of the journals replied that they felt that their test reviews were 
"somewhat influential"; two replied that they "did not know". 

Despite the fact that most of the journal editors said that the needs 
In the field were not currently being met, three of the four journals that 
currently publish test reviews Indicated that they still plan to publish 
only about the same number of teat reviews; only one Indicated plans to 
Increase the number of test reviews It will publish. On the question of 
whether a new test reviewing journal would have any eflect on plans of 
current Journals that publish test reviews, two of the five that replied 
to that question Indicated that they would publ ish "fewer test reviews" 
under those circumstances, two Indicated "no change", and one Indicated 
they would publish "more test reviews". 

While journal editors generally Indicated that the current numbers of 
test reviews are not meeting the needs In the field, but also generally 
Indicated no Intention to Increase the number of test reviews being published, 
five of seven replying journal editors Indicated that they thought their 
readers were as equally Interested In test reviews as in other content (even 
though teat reviews currently occupy only a small proportion of the space 
available). Two journal editors Indicated that their readers were more or 
much more Interested In other content than In test reviews. 

Table 15 also shows the percent of journal editors responding "yes" to 
each of seven questions describing possible problems they may havs had In 
scheduling test reviews. As the results indicate, the problem that was respon- 
ded to positively by most of the eight journal editors was that of "obtaining 
research data on tests from publfe hers", closely followed by "obtaining research 
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Table 15 

Responses to additional questions by editors of journals that did review tests 



ji 

1. would It be consistent with the policies of your organiza- 
tion to give Che Inter-Association Council on Test 
Reviewing permission to copy and/or reprint test reviews 
that have appeared in your journal for: 



a, distribution to test users on request 

yes 

no 

cannot say at this time 

b. sale to test users 

yes 

no 

cannot say at this time 



3 

0 

3 

2 

1 

3 



2, would your organization consider selling or giving reprints 
of future test reviews to the Xnter-Assoclatlcn Council 
on Test Reviewing for subsequent: 



a* distribution to test users on request 

yes 3 
no 0 
cannot say at this time 3 

b. sale to test users 

yes 2 
no 1 
cannot say at this time 3 

How influential do you feel your test reviews are on test 
selection decisions made by your readers? 

very influential 0 
somewhat influential 4 
of little influence 0 
not influential 0 
do not know 2 



X 



50.0 

0.0 

50.0 

33.3 

16.7 

50.0 



50.0 

0.0 

50.0 



33.3 

16.7 

50.0 



0.0 

66-7 

0.0 

0.0 

33.3 



What are your future plans concerning test reviews 

emphasize them more 1 
publish about the same number 3 
de-emphaslze them 0 
get out of the business of test reviewing altogether 0 



If a new Journal which was devoted exclusively to test reviews 
were startedj what would you most likely do? 

stop publishing test reviews 0 
publish fewer test reviews 2 
no change— continue publishing about the same number 2 
publish more test reviews 1 



25.0 

75.0 
0.0 
0.0 



0.0 

40.0 

40.0 

20.0 



O 
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Table 15 (continued) 



How Interested do you think readers of your Journal are in 
test reviews ns compared to the other content in your 
Journal? 

much more interested in test reviews than other con- 
tent 

more interested in test reviews 
equally Interested in teat reviews and other 
content 

more interested in other content than test reviews 
much more interested in other content than test 
reviews 



N % 

0 0.0 

0 0.0 

5 71.4 

1 14.3 

1 14.3 



Following are some problems you might have had in scheduling 
test reviews. Please indicate for each statement below 
whether or not it has been a problem for your journal. 

1. obtaining tests from publishers for review 

2. obtaining research data on tests from publishers 

3. obtaining research data on tests from the 

literature 

4. identifying competent reviewers 

5. obtaining the cooperation of test reviewers to 

review tests 

6. obtaining reviews from reviewers in time to meet 

publication deadlines 

7. knowing what tests your readers are Interested 

in having reviewed 



N %Yes 

7 42.9 

6 66.7 

5 60.0 

6 33.3 

8 37.6 

7 14.3 

7 14.3 
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data on testa from the literature". Almost half of the editors indicated 
that they had problems in "obtaining tests from publishers for review", while 
only one of the responding editors indicated that their problems included 
knowing "what tests the readers were interested in having reviewed" or 
'obtaining reviews from the reviewers in time to meet publication deadlines". 
On a comparative basis, the most serious problems were identified as numbers 
1 and 3 with one of the respondents indicating that number 5 was his most 
serious problem. 

The Journal editors were also asked to indicate whether IACTR's assis- 
tance in solving some of the problems would help the journal publish more 
reviews, and 71.4% (or five of the seven journal editors) responded affirma- 
tively. Four journal editors indicated other specific items of assistance 
that IACTR could provide in the area of test reviewing. The comments re- 
lating to additional services IACTR could provide Journals are aa follows: 
"Encourage different tvpos of reviews, that is research vs. school use of 
tests": "Provide reviews"* "Number one above la significant and if publishers 
become active members of IACTR perhaps problems would be resolved"; and 
"Need help in identifying competent reviewers in a variety of areas." 

Several journal editors also provided additional comments on the open- 
ended questions provided in the journal survey. These comments are repro- 
duced in Appendix C. 

A number of journals were identified as those which did not publish 
test reviews for their readers. Table 16 shows the response distributions 
to the questions asked of these journals. According to the journal editors, 
the major reason why their journals did not publish test reviews was that 
"There were too rany other pressures for publication of the regular content 
of our journal": 44.5% of those journal editors responding positively to one 
or more of the questions Indicated that to be the major reason why their 
journal did not publish test reviews. When each question was asked separately, 
71.4% responded affirmatively to question 4; the lowest percentage of affirm- 
ative response was to question 2. In general, the journal editors did not 
feel that their lack of publication of test reviews was because the readers 
were not Interested in tsst reviews, because they had insufficient funds, 
because it was too difficult to obtain materials, or too difficult to find 
competent reviewers. 



Insert Table 16 about here 



The final question asked of the journals not reviewing tests concerned 
whether IACTR's activities would assist them in developing a program of 
publishing test reviews. Slightly over 30% of the respondents indicated that 
they would be more likely to publish reviews if IACTR provided assistance 
both in obtaining teats for review and identifying reviewers. However, 43.8% 
of the journal editors indicated that regardless of what IACTR did they would 
not be likely to publish test reviews. 
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Table 16 

Responses to addi tional questions by editors of j ournals that did not review teats 



JSf 

For each statement below, Indicate whether or not It Is a 
reason why your Journal does not publish test reviews! 



1* the readers o£ the journal aren't Interested 

In test reviews 13 

2. lack of an appreciable number of tests In areas 

of concern to our readers 13 

3. other sources are providing enough test reviews 

to meet the needs 14 

4* there aro too many other pressures for publication 

of the regular content of our journal 14 

5. lack of sufficient funds 13 

6. it Is too difficult to obtain materials for review 13 

7. it is too difficult to find competent reviewers 13 

Would your journal be more likely to publish test reviews If _N 
the Inter-Assoclatlon Council on Test Reviewing provided ... 

assistance both In obtaining tests for review and 

In identifying reviewers 5 

assistance only In obtaining teste for review 2 

assistance only In identifying test reviewers 1 

other assistance 1 

none cf the above 7 



Vies 



23.1 



7.7 



42.8 

71.4 

15.4 
23.1 

38.5 



7. 



31.2 

12.5 

6.2 

6.2 

43.8 
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Suggestions of the respondents 



Appendix D contains the suggestions of the respondents derived from the 
open-end responses on the test user survey questionnaire. The responses are 
organized according to four general headings as follows: 1) general sugges- 

tions, 2) suggestions concerning test reviewers; 3) suggestions on the nature 
of test reviews, and 4) suggestions on the dissemination of test reviews. 

Many of these comments provide valuable insights into the problems of test users 
as well as important suggestions for the improvement of test reviewing proce- 
dures and services. 
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Summary and Implications 

Problems abound In test reviewing* About 00% of test users said that 
more test reviews are needed; 957* of journal editors agreed* Different 
organizational subgroups expressed different feelings about the nature of 
current test reviews; some groups said they were "too practical" while some 
felt they were "too technical". Only about half of those test users surveyed' 
were satisfied with the "depth" of current test reviews. Clearly, there is 
a need for different kinds of test reviews, more test reviews, and test 
reviews tailored to different audiences* 

Test reviews are not being made available enough to many audiences* 

Only one In three test users felt that he could get access to reviews when 
he needed them* Test reviews are also not being produced frequently enough* 
About eight out of ten test users are dissatisfied with the frequency of 
test reviews* There is clearly a need for new programs and mechanisms for 
the dissemination of test reviews* 

What directions should these new programs follow? Seven out of ten 
test users and journal editors said more test reviews are needed; nine out 
of ten said different types of test reviews are needed; only six of ten 
said that more test bibliographies are needed* Of these three general 
classes of needs, different types of test reviews is clearly the most 
urgent need. 

Nine out of ten of test users and journal editors supported the need for 
"comprehensive and integrative", "comparative" and "topical" reviews; different 
organizations expressed different needs* Journal editors disagreed with test 
users on "reviews of new tests"; three-fourths of editors felt these to be 
important, compared to only six of ten test users* These data imply two 
conclusions; 1) different kinds of reviews need to be developed and aimed at 
different populations; and 2) journal editors perceive only some of these 
needs accurately* 

To focus distribution of reviews where they are needed, new approaches 
to dissemination are required, with different kinds of dissemination appro- 
priate for different groups* At least six of ten test users felt that test 
reviews should be disseminated in a variety of ways, including more reviews 
in existing journals, separates available by mail, separate subject matter 
publications and a separate periodical publication* Journal editors, however, 
tended to disagree with test users on how reviews should be disseminated* 

About two of every three test users said they would subscribe to at 
least one of three new methods of dissemination of reviews, including a 
reprint service, a review subscription service and a quarterly Journal* In 
general, responses differed among the organizational subgroups, reflecting 
their different needB for test reviews* The data suggest, however, that new 
approaches to the dissemination of test reviews are likely to bo well received 
by test users, and that a large market exists for new reviewing services* 

Test users also supported new procedures for producing test reviews* 

While many indicated that reviews should be done by independent experts, 
four of ten supported the idea of a panel of reviewers for all tests, while 
about eight of ten test users felt that reviews could be produced-.by a panel 




Summary and Implications 
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o£ reviewers constituted to review specific tests* These data suggest that 
some changes may also be desirable In the way test reviews are being done* 

In general, the surveys support the contention that Important changes 
are needed in the field of test reviewing* Not only should new ways of 
producing reviews be developed, but new kinds of reviews are needed, tailored 
to different audiences, as well as Innovative ways of making these new reviews 
readily available to those who need them, when they need them* Specifically, 
these data suggest that IACTR should concentrate on the development of 
comparative, comprehensive and Integrative and topical reviews, tailored for 
specific audiences; should explore the possibilities of using panels of 
reviewers for specific tests; and should attempt to implement an integrated 
dissemination approach Including a quarterly Journal of reviews, with an 
associated review subscription service for more focused distribution, as 
well as distributing reprints of single reviews or review packages by mail- 
order service* In these ways, IACTR. can begin to take Important steps In 
the solution of the pressing problems evident .in test reviewing* 




Appendix A 



Table A-l 

Highest Academic Degree Held by Respondents) in Percentages 



Group 


N 


None 


Bachelor's 

Degree 


Master's 

Degree 


Doctor V 
Degree 


Total Group 


1)116 


0.1 


- 1.2 


37.8 


60.9 


AERA, Div. D 


109 


0.0 


0.0 


16.5 


03.5 


ARCA 


103 


0.0 


1.9 


57.3 


40.8 


ASCA 


100 


0.0 


3.0 


90.0 


7.0 


AMEG 


125 


0.0 


0.8 


72.8 


26.4 


NVGA 


127 


0.8 


3.2 


70.1 


26.0 


SPTPA 


103 


0.0 


1.9 


22.3 


75.8 


APA) Div, 12 


84 


0.0 


0.0 


77.1 


92.9 


APA, Div. 14 


95 


0.0 


1.0 


9.5 


89.5 


APA, Div. 15 


88 


0.0 


0.0 


8.0 


92.0 


APA, Div, 16 


83 


0.0 


0.0 


22.9 


77.1 


APA, Div. 17 


99 


0.0 


0.0 


31.1 


88.9 
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Table A- 2 

Primary Type of Work of Respondents, In Percentages 



Group 


N 


Teaching 


Research 


Clinical 

Eval. 


Educational 

Eval. 


Other 


Multiple 

Response 


Total Group 


1,120 


13.8 


9.3 


10.4 


11.9 


27.1® 


27.5 


AERA, Dlv. D 


109 


21.1 


27.5 


0.0 


11.0 


8.3 


32.1 


ARCA 


103 


12.6 


4.8 


16.5 


5.8 


32.0 


28.2 


ASCA 


103 


3.8 


0.0 


1.9 


20.4 


57.3 


14.6 


AMEG 


127 


7.1 


4.7 


4.7 


24.4 


37.8 


21.2 


NVGA 


128 


7.0 


1.6 


3.9 


21.9 


44.5 


21.1 


SPTPA 


104 


8.6 


7.7 


34.6 


1.0 


11.5 


36.5 


APA, Dlv. 12 


84 


20.2 


3.6 


21.4 


2.4 


14.3 


38.1 


APA, Dlv. 14 


96 


15.6 


21.9 


2.1 


2.1 


34.4 


24.0 


APA, Dlv. IS 


84 


25.0 


21.4 


8.3 


4.8 


10.8 


29.8 


APA, Dlv* 16 


83 


10.8 


4.8 


13.2 


16.9 


10.8 


43.4 


APA, Dlv. 17 


99 


24.2 


7.1 


12.1 


12.1 


23.2 


2 1.2 




T able A-3 

Pr incip al Place of EBploymen of Respondents, in Percentages 
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Table A- 4 

Age and Sex of Respondents 



Group 






Age 






Sex 




N 


Q1 


Median 


Q* 


N 


i Male 


X Feoale 


Total Group 


1,113 


38 


45 


53 


1,129 


75.3 


24.7 


ABRA, Dlv. D 


108 


36 


42 


49 


109 


82.6 


17.4 


A RCA 


101 


35 


45 


50 


103 


77.7 


22.3 


ASCA 


101 


35 


44 

„ * 


50 


102 


59.8 


40.2 


AHBO 


127 


34 


41 


49 


130 


59.2 


40.8 


NVGA 


128 


36 


43 


52 


130 


68.5 


31.5 


SPTPA 


101 


37 


46 


53 


104 


69.2 


30.8 


A PA, Dlv, 12 


84 


41 


48 


58 


84 


79.8 


20.2 


APA, Div. 14 


97 


41 


50 


57 


97 


96.9 


3.1 


APA, Dlv, 15 


88 


44 


49 


55 


89 


86.5 


13.5 


A PA, Dlv. 16 


81 


42 


49 


58 


83 


67.5 


32.5 


APA, Dlv, 17 


97 


40 


46 


56 


98 


88,8 


11.2 




Types of Assessaent Instruments Used by Respondents 
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Percentage Distribution* of Type of Measuring Instruments Most Often Used by Respondents 
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Appendix B 



Journals cooperating in journal editors survey 



American Educational Research Journal 
American Journal of Mental Deficiency 
British Journal of Educational Psychology 
Child Development 
Journal of Applied Psychology 

Journal of Child Psychology, Faychlatry and Allied Disciplines 

Journal of Clinical Psychology 

Journal of College Student Personnel 

Journal of Consulting and Clinical Psychology 

Journal of Counseling Psychology 

Journal of Educational Measurement 

Journal of Educational Psychology 

Journal of Genetic Psychology 

Journal of Projective Techniques and Personality Assessment 

Occupational Psychology 

Perceptual and Motor Skills 

Personnel end Guidance Journal 

Psychonomlc Science 

Pevlev of Educational Research 

The Clinical Paychologlst 

Vocational Guidance Quarterly 




Appendix C 

Comments of Journal Editors 



Each issue of a test review periodical might be devoted to a test or type 
of test, with articles devoted to: (a) a technical review, giving manual-typw 

Information and description; (b) a "survey of literature" paper; (c) several 
additional papers, each one on its use in a different setting or problem area, 
e.g,, research use, clinical use, use with children, use with Institutional 
populations, etc,; (d) one focusing on measurement problems, 

I feel Idea of Journal makes sense. Perhaps a quarterly. It would oof: 
seem unreasonable for this journal to perhaps carry reprints of the reviews 
originally carried In the publications aimed at specific segments of the 
test using professionals. 

I feel that test reviewing is an official function of the American 
Psychological Association. Test evaluation Is too big a Job for private 
resources. 

This Journal is governed by policies set by policies set by Publication 
Board. It Is not likely to be one of the APA Journals to move In this area. 
But Publication Board will probably support some moves In this direction. 

Our Board of Advisory Editors for this journal has reviewed the Issue of 
test reviewing. It Is our feeling that while there Is a need for more fre- 
quent reviews beyond that provided by Buros and other sources, there was very 
little agreement that the Journal should embark upon setting up a test 
review section. As 1 have Indicated earlier, we have on occasion considered 
favorably a review of a test area or a specific test, where such a review 
obviously made a significant contribution to new knowledge. 

The quality of test reviews need Improving, Our readership Is not highly 
concerned with testlt.g and those who are can get this service from other 
Journals. I would not be opposed to supporting quality teat review In our 
Journal when appropriate to our readership. 

My Journal, has a set of charter "empirical studies" which virtually 
excludes the possibility of publishing test reviews. 

I have been In favor of some kind of coordinates activities since the 
formation of the Council and Its predecessor organisation. Despite my own 
special Interest In testing, I am finding It a little difficult to get rol- 
ling on a iest review format for our Journal. 1 would probably want to 
appoint a teat review editor, but he would face some of the same problems. 

My burden would be lightened appreciably If IACTR would set up some coordi- 
nates and consultative services. Perhaps a meeting of journal editors only 
would be a helpful step* 

Oenerally, practising psychologists tend to adopt tests which neet pro- 
fessional problems in assessment, treatment etc all too uncritically. They 
have pressing and imsediata pressing problems concerning which they have to 
make an lmedlate decision. However, if new testa could be objectively 
criticised, and evaluated, compared with other ccmparable testa and such 
findings clearly auHsarlaed (a sort of consumer's association publication) 
in journals and/or separata classified publications, this might assist the 
adoption of higher standards and encourage production of good testa while 
discouraging substandard efforts. 
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Appendix C, cont. 



If we don't improve the use of test results— it will be like the goose 
who laid golden eggs— dead. The problems are more in the mis-uae of the 
results than in the tests. Another area that needs assessment is the non- 
cognitive and non academic area. Most of the available (and reliable) 
tests are in the academic areas. 

Encourage different types of reviews, that is research vs. school use 
of tests. 

Sorry, to give an unhelpful reply— but this is not an area which we deal 
with. Research papers may in fact often include material which could be 
described as "test reviews", but we publish on the strength of the research 
contribution, not because they are reviews of teats. 

Almost all reviews are incompetent. Test authors and publishers should 
have space to comment on stupid, unperceptlve, even Inaccurate statements 
by reviewers. We gavo up on Buros years ago. Others am not much better on 
the average. .... Let me repeat. The adequate teat reviewer must have: 

1) extensive experience with the test; 2) training in traditional approaches 
to test construction, standardization and validation; 3 ) real sophistication 
in the philosophy of science as It applies to measurement in all sciences. 
Very few people like this, so almost all reviews poor. Test authors should 
have an opportunity to try to block (by persuasion) unfair statements by 
reviewers, and should be guaranteed journal space to reply to (or consent on) 
reviews, as a matter of fixed policy. Our journals ere the only onos which 
consistently do this. ... Please be super-careful in any steps you take— this 
is a sensitive field in which mo3t reviews are badly biased, and intellec- 
tually Incompetent (from a measurement theory point of view). We don't 
really need more journals, but more Intellectual competence. 




